AMENDMENTS TO THE CLAIMS 



1 . (Currently amended) A method for automating the optimization of search results 
displayed in a search Web page, the method comprising: 

collecting data that represents a performance of a search result, the data originating from 
at least one of a plurality of sources of performance data; 

normalizing the collected data in accordance with a relative importance of the source of 
the data; 

comparing the normalized performance data to an expected performance data for the 
search result; 

diagnosing at least one possible cause for an underperforming search result from the 
results of the comparison between the normalized performance data and the expected 
performance data ; and 

adjusting an operation of a search engine that produced the search result in accordance 
with the diagnosis to improve the search result performance , the adjustment operative to give 
greater significanc e to performance data from at least one of the sources . 

2. (Original) The method of Claim 1, wherein the at least one of the plurality of 
sources of performance data includes one of an implicit performance data, an explicit 
performance data, a human-judged performance data, a relevance verification data, and a sample 
test data. 

3. (Original) The method of Claim 1, wherein normalizing the collected data in 
accordance with a relati ve importance of the source of the data includes giving greater weight to 
the data from the more important sources and combining the data to reflect the relative 
importance of the source from which the data originated. 
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4. (Currently amended) The method of Claim 2, wherein comparing the normalized 
performance data to the expected performance data includes determining that the search result is 
underperforming when the normalized performance is below or substantially a quantified 
threshold below the expected performance. 

5. (Currently amended) The method of Claim 2, wherein the most important source 
of data is implicit performance data, and normalizing the collected data includes giving implicit 
performance data substantially greater weight when combining the data. 

6. (Original) The method of Claim 2, wherein implicit performance data is 
automatically captured when a user interacts with the search result. 

7. (Original) The method of Claim 6, wherein implicit performance data includes at 
least one of whether the user clicked on the result, a location of the result when the user clicked 
the result, and a length of time that the user browsed the result. 

8. (Original) The method of Claim 7, wherein implicit performance data further 
identifies an operation that the user performed on the result, including at least one of editing, 
e-mailing, printing, bookmarking, and copying. 

9. (Original) The method of Claim 6, wherein implicit performance data includes 
captured data that has been aggregated across multiple interactions with the search result. 

10. (Original) The method of Claim 6, wherein implicit performance data includes 
captured data that has been aggregated across multiple users interacting with the search result. 
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11. (Original) The method of Claim 2, wherein explicit performance data includes 
data obtained from user response to a search engine operator-generated inquiry about the search 
result, wherein the inquiry includes one of an on-line inquiry and a telephone inquiry. 

12. (Original) The method of Claim 2, wherein human- judged performance data 
includes data obtained from a hximan evaluation of the search result. 

13. (Original) The method of Claim 2, wherein relevance verification data includes 
data obtained from a verification test of the search result's relevance to verify whether a search 
result having a known relevance is still included in the search result generated by the search 
engine. 

14. (Original) The method of Claim 2, wherein sample test data includes data 
obtained from a test of the search result's relevance performed on a sample of a subset of users. 

15. (Original) The method of Claim 2, wherein diagnosing at least one possible cause 
for an underperforming search result includes considering at least one of whether the search 
result is no longer valid, whether the search result appears in a poor location, whether a search 
term that generated the search result is easily misspelled, whether the search term is too broad to 
generate a meaningful result, and whether a search for the search term should be constrained to a 
specific resource. 

16. (Original) The method of Claim 2, wherein adjusting an operation of a search 
engine that produced the search result in accordance with the diagnosis to improve the search 
result performance, includes modifying the search engine's search schema to change the search 
result generated for the search term, including at least one of reranking, removing, and replacing 
the search results. 

LAW OFFICES OF 
CHRISTENSEN O'CONNOR JOHNSON KINDNESS™* 
1420 Fifth Avenue 
Suite 2800 
Seattle, Washington 98101 
-9- 2066828100 

MSFT\21738AM.DOC 



17. (Original) The method of Claim 2, wherein adjusting an operation of a search 
engine that produced the search result in accordance with the diagnosis to improve the search 
result performance, includes modifying the search engine's search schema to augment a 
presentation of the search results generated for the search term, including at least one of 
highlighting, animating, enlarging, and repositioning an appearance of the search result on a 
search results web page. 

18. (Original) The method of Claim 2, wherein adjusting an operation of a search 
engine that produced the search result in accordance with the diagnosis to improve the search 
result performance, includes increasing the search engine's spellchecker tolerance. 

19. (Original) The method of Claim 2, wherein adjusting an operation of a search 
engine that produced the search result in accordance with the diagnosis to improve the search 
result performance, includes prompting the user to one of clarify or narrow the search term with 
an additional user input. 

20. (Original) The method of Claim 2, wherein adjusting an operation of a search 
engine that produced the search result in accordance with the diagnosis to improve the search 
result performance, includes temporarily adjiisting the operation of the search engine, and 
determining whether the adjustments have actually improved the search result performance 
before permanently adjusting the operation of the search engine. 

21. (Original) The method of Claim 2, wherein adjusting an operation of a search 
engine that produced the search result in accordance with the diagnosis to improve the search 
result performance, includes adjusting the operation of the search engine in real time. 
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22. (Original) The method of Claim 21, wherein adjusting the operation of the search 
engine in real time includes intercepting the search result generated by the search engine and 
modifying the search result before the search engine displays the search result to the user. 

23. (Currently amended) An automated search result optimization system that 
provides search results to a user, comprising: 

an input data representing a performance of a search result generated by a search engine 
for a search term, the input data originating from a plurality of sources; 

a data collection process to collect the input performance data from the plurality of 
sources; 

a diagnostic process to compare a performance of the search result, as represented by the 
collected performance data, to an expected performance of the search result; and 

an adjustment process to generate an output data representing an action to automatically 
adjust an operation of the search engine whenever the search result performance c o mp a res 
unfavorably to diverges a quantified threshold from the expected performance, wherein the 
adjusted operation of the search engine is to improve the performance of the search result. 

24. (Original) The system of Claim 23, wherein the at least one of the plurality of 
sources of input performance data includes one of an implicit performance data, an explicit 
performance data, a human-j udged performance data, a relevance verification data, and a sample 
test data. 

25. (Original) The system of Claim 23, wherein the data collection process includes a 
normalizing process to normalize the collected data in accordance with a relative importance of 
the source of the data, and further wherein normalizing includes giving greater weight to the data 
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from the more important sources and combining the data to reflect the relative importance of the 
source from which the data originated. 

26. (Currently amended) The system of Claim 25, wherein the diagnostic process 
compares the performance of the search result as represented by the normalized collected 
performance data to an expected performance of the search result; and further, wherein the 
comparison is unfavorable when the performance is lower than or substantially lower 4han a 
quantified threshold value below the expected performance. 

27. (Currently amended) The system of Claim 25, wherein the most important source 
of data is implicit performance data, and normalizing the collected data includes giving implicit 
performance data substantially greater weight when combining the data. 

28. (Original) The system of Claim 27, wherein implicit performance data is 
automatically captured when a user interacts with the search result. 

29. (Original) The system of Claim 28, wherein implicit performance data includes at 
least one of whether the user clicked on the result, a location of the result when the user clicked 
the result, and a length of time that the user browsed the result. 

30. (Original) The system of Claim 29, wherein implicit performance data further 
includes data representing an identification of an operation that the user performed on the result, 
including at least one of editing, e-mailing, printing, bookmarking, and copying. 

3 1 . (Original) The system of Claim 28, wherein implicit performance data includes 
captured data that has been aggregated across multiple interactions with the search result. 
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32. (Original) The system of Claim 28, wherein implicit performance data includes 
captured data that has been aggregated across multiple users interacting with the search result. 

33. (Original) The system of Claim 24, wherein explicit performance data includes 
data obtained from user response to a search engine operator-generated user satisfaction survey 
about the search result. 

34. (Original) The system of Claim 24, wherein human-judged performance data 
includes data obtained from a human evaluation of the search result. 

35. (Original) The system of Claim 24, wherein relevance verification data includes 
data obtained from a verification test of the search result's relevance to verify whether a search 
result having a known relevance is still included in the search result generated by the search 
engine. 

36. (Original) The system of Claim 24, wherein sample test data includes data 
obtained from a test of the search result's relevance performed on a sample of a subset of users. 

37. (Original) The system of Claim 24, wherein the diagnostic process further 
includes diagnosing at least one possible reason why the performance of the search result 
compares unfavorably to the expected performance, including considering at least one of 
whether the search result is no longer valid, whether the search result appears in a poor location, 
whether a search term that generated the search result is easily misspelled, whether the search 
term is too broad to generate a meaningful result, and whether a search for the search term 
should be constrained to a specific resource. 
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38. (Original) The system of Claim 23, wherein the adjustment process to generate 
an output data representing an action to automatically adjust an operation of the search engine, 
includes generating output data that represents an action to modify the search engine's search 
schema, wherein the modified search schema changes the way the search engine generates a 
search result for a search term, including at least one of reranking, removing, and replacing the 
search result . 

39. (Original) The system of Claim 23, wherein the adjustment process to generate 
an output data representing an action to automatically adjust an operation of the search engine, 
includes generating output data that represents an action to modify the search engine's search 
schema, wherein the modified search schema augments the way the search engine presents the 
search result for a search term, including at least one of highlighting, animating, enlarging, and 
repositioning an appearance of the search result on a search result Web page. 

40. (Original) The system of Claim 23, wherein the adjustment process to generate 
an output data representing an action to automatically adjust an operation of the search engine, 
includes generating output data that represents an action to increase the search engine's 
spellchecker tolerance. 

41. (Original) The system of Claim 23, wherein the adjustment process to generate 
an output data representing an action to automatically adjust an operation of the search engine, 
includes generating output data that represents an action to prompt the user to one of clarify or 
narrow the search term with an additional user input. 

42. (Original) The system of Claim 23, wherein the adjustment process to generate 
an output data representing an action to automatically adjust an operation of the search engine, 
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includes generating output data that represents an action to temporarily adjust the operation of 
the search engine, and to further determine whether the temporary adjustment has actually 
improved the search result performance before generating an action to permanently adjust the 
operation of the search engine. 

43. (Original) The system of Claim 23, wherein the adjustment process to generate 
an output data representing an action to automatically adjust an operation of the search engine, 
includes generating output data that represents an action to adjust the operation of the search 
engine in real time. 

44. (Currently amended) The system of Claim [[42]] 43 herein the action to adjust 
the operation of the search engine in real time includes an action to intercept the search result 
generated by the search engine and to further modify the search result before the search engine 
displays the search result to the user. 

45. (Currently amended) One or more computer-accessible media having instructions 
stored on the media for facilitating the automated optimization of a search result in a search 
result user interface, the instructions comprising: 

collecting data that represents a performance of a search result from at least one of a 
plurality of sources o f performance data; 

normalizing the collected data in accordance with a relative importance of the source of 
the data; 

comparing the normalized performance data to an expected performance data for the 
search result; 

diagnosing at least one possible reason why the sear c h result performance compares 
unfavorably to the - expectc d performance cause for an underperforming search result based on 
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the comparison between the normalized performance data and the expected performance data ; 
and 

adjusting an operation of a search engine that produced the search result in accordance 
with the diagnosis to improve the search result performance , the adjustment operative to give 
greater significance to performance data from at least one of the sources . 

46. (Original) The computer-accessible media of Claim 45, wherein the at least one 
of the plurality of sources of performance data includes one of an implicit performance data, an 
explicit performance data, a human-judged performance data, a relevance verification data, and a 
sample test data. 

47. (Original) The computer-accessible media of Claim 45, wherein the instruction to 
normalize the collected data in accordance with a relative importance of the source of the data 
includes instructions to give greater weight to the data from the more important sources and to 
combine the data to reflect the relative importance of the source from which the data originated. 

48. (Currently amended) The computer-accessible media of Claim 45, wherein the 
instruction to compare the normalized performance data to the expected performance data 
includes an instruction to determine that the search result is underperforming when the 
normalized performance is below or substantially a quantified threshold value below the 
expected performance. 

49. (Currently amended) The computer-accessible media of Claim 47, wherein the 
most important source of data is implicit performance data, and the instruction to normalize the 
collected data includes an instruction to give implicit performance data su b stantially greater 
weight when combining the data. 
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50. (Original) The computer-accessible media of Claim 46, wherein implicit 
performance data is data that is automatically captured by a search engine when a user interacts 
with a search result. 

5 1 . (Original) The computer-accessible media of Claim 50, wherein implicit 
performance data includes data that represents at least one of whether the user clicked on the 
result, a location of the result when the user clicked the result, and a length of time that the user 
browsed the result. 

52. (Original) The computer-accessible media of Claim 50, wherein implicit 
performance data further identifies an operation that the user performed on the result, including 
at least one of editing, e-mailing, printing, bookmarking, and copying. 

53. (Original) The computer -accessible media of Claim 50, wherein implicit 
performance data includes captured data that has been aggregated across multiple interactions 
with the search result. 

54. (Original) The computer-accessible media of Claim 50, wherein implicit 
performance data includes captured data that has been aggregated across multiple users 
interacting with the search result. 

55. (Original) The computer-accessible media of Claim 46, wherein explicit 
performance data includes data obtained from user response to a search engine operator- 
generated inquiry about the search result, wherein the inquiry includes one of an on-line inquiry 
and a telephone inquiry. 
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56. (Original) The computer-accessible media of Claim 46, wherein human-judged 
performance data includes data obtained from a human evaluation of the search result. 

57. (Original) The computer-accessible media of Claim 46, wherein relevance 
verification data includes data obtained from a verification test of the search result's relevance to 
verify whether a search result having a known relevance is still included in the search result 
generated by the search engine. 

58. (Original) The computer-accessible media of Claim 46, wherein sample test data 
includes data obtained from a test of the search resulf s relevance performed on a sample of a 
subset of users. 

59. (Original) The computer-accessible media of Claim 45, wherein the instruction to 
diagnose at least one possible reason why the search result performance compares unfavorably to 
the expected performance includes an instruction to determine at least one of whether the search 
result is no longer valid, whether the search result appears in a poor location, whether a search 
term that generated the search result is easily misspelled, whether the search term is too broad to 
generate a meaningful result, and whether a search for the search term should be constrained to a 
specific resource. 

60. (Original) The computer-accessible media of Claim 45, wherein the instruction to 
adjust the operation of the search engine that produced the search result in accordance with the 
diagnosis, includes an instruction to modify the search engine's search schema to change the 
search result generated for the search term, wherein to change the search result includes at least 
one of to rerank, remove, and replace the search result. 
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6 1 . (Original) The computer-accessible media of Claim 45, wherein the instruction to 
adjust the operation of the search engine that produced the search result in accordance with the 
diagnosis, includes an instruction to modify the search engine's search schema to augment a 
presentation of the search result generated for the search term, wherein to augment the 
presentation includes at least one of to highlight, animate, enlarge, and reposition a display of the 
search result on a search result Web page. 

62. (Original) The computer- accessible media of Claim 45, wherein the instruction to 
adjust the operation of the search engine that produced the search result in accordance with the 
diagnosis, includes an instruction to increase the search engine's spellchecker tolerance. 

63. (Original) The computer-accessible media of Claim 45, wherein the instruction to 
adjust the operation of the search engine that produced the search result in accordance with the 
diagnosis, includes an instruction to prompt the user to one of clarify or narrow the search term 
with an additional user input. 

64. (Original) The computer- accessible media of Claim 45, wherein the instruction to 
adjust the operation of the search engine that produced the search result in accordance with the 
diagnosis, includes an instruction to temporarily adjust the operation of the search engine, and an 
instruction to further determine whether the adjustments have actually improved the search result 
performance before performing an instruction to permanently adjust the operation of the search 
engine. 

65. (Original) The computer-accessible media of Claim 45, wherein the instruction to 
adjust the operation of the search engine that produced the search result in accordance with the 
diagnosis, includes an instruction to adjust the operation of the search engine in real time. 
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66. (Original) The computer-accessible media of Claim 65, wherein the instruction to 
adjust the operation of the search engine in real time includes an instruction to intercept the 
search result generated by the search engine and a further instruction to modify the search result 
before the search engine displays the search result to the user. 
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